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RAYMOND J. UHALDE 
Deputy Assistant Secretary for 
Employment and Training 


FROM: 1 


fJOUN JJ GETEK 
^Assistant Inspector General 
for Audit 


SUBJECT: 


Audit of the Job Training Parmership Act 
Out-of-School Youth Pilot Demonstration Grant 
Program for Four Final Round Pilot Grantees 
Final Audit Report No. 06-01-002-03-340 



The attached subject final report is submitted for your resolution action. We request a 
response to this report within 60 days. 

If you have any questions concerning this report, please contact John F. Riggs, Regional 
Inspector General for Audit, Dallas, at (214) 767-6980. 
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Executive Summary 



BACKGROUND 

The Department of Labor (DOL), Employment and Training Administration (ETA), awarded 
its final round of Job Training Partnership Act (JTPA), Tide IV, PUot and Demonstration 
Grant Program funds for the Youth Opportunity Area (YOA), Out-of-School Youth (OSY) 
program in April 1999. These final round grants were awarded to five cities ~ Denver, San 
Diego, Oakland, Detroit, and Baltimore ~ to improve employment prospects for out-of-school 
youth,' ages 16 through 24, in each city’s designated low income neighborhood. We reviewed 
the program structure and activities provided a sample of youth for four of the five final round 
grantees. Oakland, which did not receive an operational grant under the Workforce Investment 
Act (WIA) Youth Opportunity (Yo!) program, was not included in our review. 



This audit was a followup to the OIG’s March 22, 2000, report (No. 06-00-002-03-340) on the 
first round YOA OSY pilot demonstration grantees’ performance. 

AUDIT RESULTS 

Enrollees’ participation rates have increased from the first round grantees; yet, almost 
half of the enrollees were no longer in contact with the program at the time of our 
fieldwork. 

The final round pilot grantees we audited better addressed most of the youths’ needs than we 
found in our first audit. Services to high school dropouts also increased, and most of their 
needs were addressed with educational or vocational training activities. 

Although an emphasis of YOA was that program staff was supposed to function as mentors and 
counselors rather than routine case managers, case files evidenced that fewer than half of the 
sampled youth received bona fide counseling. However, our limited participant interviews 
indicated more bona fide counseling and mentoring occurred than was evident from the case 
files. 

If counseling/mentoring is to be reported as a bona fide activity, better documentation is 
needed to support this activity, especially in light of the amount of grant funds that are used for 
this service. 

In-school-youth services/outcomes may distort out-of-school youth demonstration grants’ 
results. 

Of our participant sample, 27 percent were attending school (in-school) at time of enrollment 
in the out-of-school demonstration grant programs, including one grantee which operated a 
specific in-school program which targeted youth 14 through 15 years of age who were 
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currently in school but at risk of dropping out. In our opinion, any performance results 
reported on the in-school youth could potentially inflate performance reporting for the YOA 
OSY program; e.g., an "attending school" activity could be misinterpreted to indicate the 
program influenced a dropout to return to school. 

Performance reporting systems were improved, but documentation available to support 
those activities/outcomes is deficient. 

All of the sites had automated management information systems (MIS) in place either during or 
subsequent to our fieldwork to capture the necessary data to report the youths’ activities and 
outcomes. However, the level of evidence of enrollees’ activities, including the extent of their 
participation, still needs to be increased if the youths’ activities and outcomes are to be 
reported in accordance with the Workforce Investment Act (WIA) and the Government 
Performance and Results Act (GPRA) reporting standards. 

Recommendations : 

We recommend the Assistant Secretary for Employment and Training develop a system of 
management controls that provides reasonable assurance that the competitive WIA-funded Yo! 
Program will result in substantial and demonstrable youth educational gains and vocational 
skills attainment in the grants’ targeted areas. Specifically, we recommend ETA. 

• Develop documentation standards and guidelines for grantees to help educate assigned 
staff and reinforce ETA expectations for minimum case file evidence of youth activities 
and accomplishments/outcomes. 

• Assure that separate performance reporting is maintained for Yo! activities, 
accomplishments, and outcomes for in-school as opposed to out-of-school-youth. 

• Issue guidance to all Yo! program grantees encouraging the grantees to: 

• Assess each participant’s needs when enrolled, and as necessary throughout 
program participation. This should be a complete formal assessment, including 
participants input into what needs to be done for them to keep them actively 
enrolled until all substantial needs are met. 

• Provide a wide array of quality services, particularly more vocational training, 
to address the participants’ identified needs, either through WIA funds or other 
available sources (e.g.. Pell grants); and 

• Develop and implement performance incentives to high risk program dropouts 
for major accomplishments. 
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ETA’S Response to Our Draft Report and Auditor’s Conclusion 

The Deputy Assistant Secretary for Employment and Training was pleased with the reported 
improvements in both enrollee participation rates and performance reporting systems. 
However, he stated that service providers must continue to improve the documentation of 
services participants receive. He also indicated that ETA would work with the grantees to 
improve the documentation and to implement the other recommendations. 

We concur with the Deputy Assistant Secretary’s response. 



O 
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Background 

The ETA issued Job Training Partnership Act (JTPA), Title IV, Youth Opportunity Area (YOA), 
Out-of-School (OSY) pilot demonstration grants to 11 cities to serve youth in high poverty targeted 
areas. These grants were issued in three rounds: three grants in round one; three in round two; and 
five in round three. 

The YOA OSY pilot demonstration grants were to provide: 

... a structured set of initiatives focused sharply on getting out-of-school youth 
ages 16-24 into long-term employment at wage levels that will prevent future 
dependency. The various programs funded under the grant should constitute a 
coherent strategy for serving large numbers of neighborhood youth and raising the 
employment rate of out-of-school youth in the area up to 80 percent. (Source: 

ETA’s Request for Proposal) 

The YOA OSY demonstration grant program was intended to provide services above and beyond 
the formula JTPA youth programs. This program also emphasized coordination between all 
services available to youth (e.g., community-based organizations (CBOs), social services, 
education, etc.). In addition, this program was to emphasize intensive counseling and mentoring. 
Followup services were required and intended to support youth through placement in either 
employment or long-term educational programs. 

The goals for the final round of YOA OSY demonstration grants were to expand employment, 
education, and training opportunities for out-of-school youth ages 16 through 24, with priority 
given to high school dropouts. These goals differ somewhat from the initial round of 
demonstration grants in that they include education and trauiing opportunities for target area youth, 
not solely focusing efforts on employment. However, the goal of 80 percent OSY employment rate 
was still noted in ETA’s Request for Proposal (RIT). 

Although serving in-school youth was added as a component of the WIA-fimded Yo! competitive 
operational grants awarded in the spring of 2000, the final round JTPA-fimded OSY pilot 
demonstration grants were intended to serve out-of-school youth. It was our imderstanding that, 
while in-school youth were not to be denied services, they were to be served with nongrant funds, 
and in-school youth performance statistics were not to be included with the intended YOA OSY 
population. 

On March 22, 2000, the Office of Inspector General (OIG) issued Report No. 06-00-002-03-340 on 
the results of our audit of the first round YOA demonstration grants. In response to the report, 
ETA agreed the YOA demonstration grant programs needed improvement. However, ETA also 
commented on these pilot grants’ experimental nature and the lessons learned from them. ETA 
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indicated that the final round pilot grantees have a much stronger management information system 
than the initial three sites and stated the initial three sites have improved since the period we 
examined. 

This report presents the results of the performance audit we conducted on four final round grantees 
as a followup to our earlier findings related to the first round YOA grant program. 



O ut-o f-School Youth Demonstration Grant Audit - Final Round Grantees 

Objectives, Scope, and Methodology 

Our primary audit objectives were to determine whether the OSY pilot demonstration grant 
program was operating as proposed and whether the youth were being served as intended. Other 
subobjectives included: 

• Are the grantee’s methods of serving youth focused toward providing adequate training 
and educational services which should result in increasing employment rates? 

• Is the counseling/mentoring activity merely case management or bona fide personal, 
ongoing support for the youth through their educational and employment process? 

• Does the grantee have adequate capability to prepare performance reports that account 
for the activities and outcomes of the enrolled youth? 

Each of the four final round YOA OSY pilot demonstration grantees included in this audit received 
about $2.25 million for the period April 5, 1999, through October 5, 2000. The 4 sites included in 
this audit were selected because they were among the 36 competitive grantees that received WIA 
Yo! operational grants totaling $225 million in the spring of 2000. For each of the four sites, we 
selected an audit sample of 25 participants from the total participants enrolled under these grants as 
of December 31, 1999. Consequently, our audit examined services and outcomes for 100 
participants, or one-third of the 305 total participants served by the four grantees as of 
December 31, 1999. Audit fieldwork began in August 2000. 

We reviewed participants’ files for the sample selected at each location to determine the training 
and counseling activities performed, and reviewed case file documents which presumably supported 
the activities and performance statistics reported to ETA. We reviewed the case files to determine 
how the grantee performed case management and enrollee followup activities. We also interviewed 
participants (14 of 100)' and examined grantees’ case file documentation to determine whether 
there was coordination between other community-based organizations (CBOs) and social service 
programs necessary to ensure the program was serving the clients’ needs. Finally, we reviewed the 
grantees’ MIS to see how they tracked activities under these grants 

We conducted our audit in accordance with Government Auditing Standards for performance audits 
issued by the Comptroller General of the United States. 

Audit procedures used to complete the review were not designed or intended to give an opinion on 
the adequacy of internal or program controls covering performance reporting at the local, state, or 

Federal level. 



'At the request of the Assistant Secretary for Employment and Training, we attempted to interview all 100 sampled 
participants but the grantees were only able to get 14 participants for these interviews. 
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Out-of'School Youth Demonstration Grant Audit - Final Round Grantees 

Audit Results 

1. Enrollees’ Participation Rates Have Increased from the First Round Grantees; Yet, Almost 
Half of the Enrollees Were No Longer in Contact with the Program at the Time of Our 

Fieldwork. 



The final round pilot grantees did a better job of addressing most of the youths’ needs than we 
found in our initial audit. Services to high school dropouts also increased (90 percent received 
some type of service); most of the dropouts’ identified needs were addressed with educational or 
vocational training activities. Yet, more than half of our 100 participant sample - enrolled as of 
December 31, 1999 - were no longer actively participating in the program as of August 2000, and 
42 percent of the youth were no longer in contact with their counselors. 

The following analyses are useful in obtaining a preliminary overview of enrollee activities and 
status obtained from the four programs for sampled enrollees through July 2000. (The enrollee 
files were obtained in August 2000). 



A. Specific enrollee activities 



From our 100 sampled enrollees: 

• 89 (89 percent) enrollees participated in an activity after enrollment (as opposed to 67 
percent in our first round pilot audit). 

• 74 (74 percent) enrollees received some educational or vocational traming (as opposed 
to 50 percent in our first round pilot audit). 



Priority of YOA OSY services was to be given to dropouts. Of the 58 dropouts in our 100 enrollee 
sample, 52 participated in one or more program services as follows: 



• 38 (73 percent) returned to school or participated in a basic skills/GED program (as 
opposed to 39 percent in the first roimd audit). 

• 17 (33 percent) participated in a vocational skills traming program (as opposed to 11 
percent in the first roimd audit). 



Table 1 provides enrollees’ educational/training activities -- excluding job referrals, job related 
tr aining , or recreational activities. In some instances, YOA OSY funds did not assist in paying for 
these activities (e.g., work experience and high school enrollment). The enrollees files frequently 
did not indicate either the length of time individuals attended school or traming or whether the 
enrollee actually completed a GED or other training program. Therefore, table 1 indicates only that 
the enrollee participated in the activities; it does not measure the level of p^icipation or completion. 
The levels of evidence we used to make the determination of enrollees’ activities is discussed in 

finding 3. 



O 
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Table 1 

Enrollee Educational/Training Accomplishments by Age Group 



Age Group 


Received no 
Educational 
or Vocational 
Training 


Educational/Training Services Received 


Attended 

High 

School 


Participated in 
Basic 

Skills/GED 

Program 


Participated 
in English as 
a Second 
Language 
Training 


Attended 

Formal 

Vocational 

Training 


18 and Under 


18% 


41% 


59% 


2% 


16% 


(51) 


(9) 


(21) 


(30) 


(1) 


(8) 


19-21 


38% 


3% 


28% 


0% 


41% 


(32) 


(12) 


(1) 


(9) 


(0) 


(13) 


22-24 


29% 


(0%) 


24% 


0% 


71% 


(17) 


(5) 


(0) 


(4) 


(0) 


(12) 


Total Sample 


26% 


22% 


43% 


1% 


33% 


(100)^ 


(26)' 


(22) 


(43) 


(1) 


1 (33) 



(The numbers in parenthesis denote the actual number of enrollees in the respective categories.) 



As shown in table 1, above: 

• One-third of the sampled enrollees received vocational training after YOA OSY 
enrollment. 

• Most of the enrollees who attended high school or attended a basic skills/GED program 
were 18 or younger. 

• Seventy-one percent of youth ages 22 through 24 participated in vocational training. 

(This represented a significant increase over the results of our first round audit, in which 
we found 24 percent of youth 22-24 attended vocational training). 



^Some enrollees had more than one activity; therefore, the activity totals of all columns exceed 100 enrollees. 
^Eleven youth received no services at all. 
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B. The grantees attempted to meet most enrollees’ identified needs 

We examined enrollees’ files to determine what specific services were being provided. Most of the 
case files contained indications of some type of assessment completed for each enrollee. This 
assessment, along with any other indication in the case file, was used to identify specific training or 
employment needs expressed by the youth. 

Of the 100 sampled enrollees, 95 had a training need identified, and 83 (87 percent) of those had that 
need addressed through some type of grant-arranged assistance. 

Table 2, below, shows how often identified enrollee needs were addressed, regardless of the amount 
of the YOA OSY-fimded assistance. For example, the need for vocational training was identified in 
61 cases but only addressed in 32 cases (52 percent). 

Table 2 

Services Provided by Age Group 
For All Sample Enrollees Where the File Identified a Need 



Age 

Group 


High 

School 


Basic Sks 
GED 


ESL 


Voc. 

Trng. 


Work 

Exper. 


Job 

Prep. 


Job 

Referrals 


Recr. 

Act. 


Other 

Trng. 


18 and 
Under 


78% 

(21 of 27) 


81% 

(29 of 36) 


100% 
(1 of 1) 


32% 

(8 of 25) 


100% 
(11 of 11) 


65% 

(28 of 43) 


59% 

(20 of 34) 


100% 
(9 of 9) 


100% 
(3 of 3) 


19-21 


33% 
(1 of 3) 


60% 

(9 of 15) 


0% 

(0 of 1) 


57% 

(13 of 23) 


100% 

(13 of 13) 


68% 

(15 of 22) 


83% 

(24 of 29) 


100% 
(12 of 12) 


100% 

(2 of 2) 


22-24 


0% 

(0 of 1) 


67% 
(4 of 6) 


0% 

(0 of 0) 


85% 

(11 of 13) 


100% 
(5 of 5) 


82% 

(9 of 11) 


76% 

(13 of 17) 


100% 
(5 of 5) 


100% 
(5 of 5) 


Overall 


71% 

(22 of 31) 


74% 

(42 of 57) 


50% 
(1 of 2) 


52% 

(32 of 61) 


100% 

(29 of 29) 


68% 

(52 of 76) 


71% 

(57 of 80) 


100% 

(26 of 26) 


100% 
(10 of 10)“ 



Although there was evidence in the case files that some form of assessment was performed for 
each enrollee, the type of evidence varied greatly and was minimal in many cases. We 
acknowledged the enrollees as needing the services they were provided, even if there was no 
specific need identified. As compared to our first audit, it appeared the grantees are focusing 
on better serving the youths’ needs. 

We based our needs analysis on any available case file documentation, since there was no 
requirement for the demonstration grant to conduct an enrollee assessment and develop a 
service strategy. For example, one site’s participant application had general questions about 



"*The "other training" category consisted of one youth attending community college, two youth participating in a 
youth leadership program, and seven youth attending support group counseling meetings. 
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what training the youth was interested in, and we gave the site credit for completing an 
assessment for the participant. 

We analyzed how the grantees addressed participants’ needs separately for those without high 
school educations (table 3) and those with high school educations or higher (table 4). 



Table 3 

Services Provided to Enrollees by Age Group 
Less Than High School Graduates 



For the 73 youth with less than a high school educational level with an identifled need» the 
following services were provided.® 



Age 

Group 


High 

School 


GED 


Vocat. 

Ed. 


Work Exp. 
Comm. Srvc 


Job Prep. - 
Work Read. 


Job 

Referrals 


Recreation 

Activities 


18 and 
Under 


78% 

(21 of 27) 


83% 

(30 of 36) 


27% 

(6 of 22) 


100% 
(10 of 10) 


62% 

(24 of 39) 


53% 

(16 of 30) 


100% 
(8 of 8) 


19-21 


33% 
(1 of 3) 


54% 

(7 of 13) 


50% 

(5 of 10) 


100% 
(4 of 4) 


75% 

(9 of 12) 


73% 

(1 1 of 15) 


100% 
(4 of 4) 


22-24 


0% 

(Oof 1) 


67% 
(4 of 6) 


78% 
(7 of 9) 


100% 
(3 of 3) 


88% 
(7 of 8) 


70% 

(7 of 10) 


100% 
(4 of 4) 


Total 


71% 

(22 of 31) 


75% 

(41 of 55) 


44% 

(18 of 41) 


100% 
(17 of 17) 


68% 

(40 of 59) 


62% 

(34 of 55) 


100% 
(16 of 16) 



As table 3 shows, the program attempted to get the youth with less than a high school 
education into needed traming. 

Youths for whom a need was identified were enrolled in vocational training at a lower rate 
than any other service. We also noted that for work experience and recreational activities few 
needs were specifically identified in the case files. However, we gave the program credit for 
addressing the need if the youth received the service, even if no need was specified. This 
resulted in the high rates of addressing the need for those two categories. 



®One youth was not assessed, thus, no needs were identified for one less than high school sampled enrollee. 

®The program also provided one youth with ESL training, and six youth with other training (consisting of 
narticipating in the site’s formal youth recruitment program and attending support groups). Due to the minimal number of 
youth participating in these activities and the fact that most of the youths’ needs were met for both categories, they were 
included in tl'is footnote 3nd not in the tEble above. 
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Table 4 

Services Provided to Enrollees by Age Group 
High School Graduates (or Equivalent) and Above 



For 26 high school graduates with an identified need, the following services were provided.’ 



Aae Group 


Basic 

SkUIs 


Vocational 

Education 


Work Exp. - 
Comm. Srvc. 


Job Prep. 
Work Read. 


Job 

Referral 


Recreation 

Activities 


18 and Under 


0% 

(OofO) 


67% 
(2 of 3) 


100% 
(1 of 1) 


100% 
(4 of 4) 


100% 
(4 of 4) 


100% 
(1 of 1) 


19-21 


100% 
(2 of 2) 


62% 

(8 of 13) 


100% 
(9 of 9) 


60% 

(6 of 10) 


93% 

(13 of 14) 


100% 
(8 of 8) 


22-24 


0% 

(OofO) 


100% 
(4 of 4) 


100% 
(2 of 2) 


67% 
(2 of 3) 


86% 
(6 of 7) 


100% 
(1 of 1) 


Total 


100% 
(2 of 2) 


70% 

(14 of 20) 


100% 
(12 of 12) 


71% 

(12 of 17) 


92% 

(23 of 25) 


100% 
(10 of 10) 



As shown above, the grantees also attempted to address the needs for those youth with an 
above high school graduate educational level. 



C. Services for dropouts have improved since our last review 

We identified 58 high school dropouts* from our 100 enrollee sample. Of these 58 youth, 52 
(90 percent) participated in some educational and vocational training activities after enrollment: 

• 9 (17 percent) returned to high school. 

• 29 (56 percent) participated in a basic skills/GED program. 

• 1 (2 percent) attended an ESL program. 

• 17 (33 percent) attended some vocational training program. 

As compared to our first round audit, vocational training and educational activities have 
increased for the dropout population in this final round audit. There was evidence in the final 
round sample that 4 of the 17 youth (24 percent) participating in vocational education 
completed the course. 



’xhe program also provided four youth with other training (consisting of one youth attending co mmuni ty college 
and three youth attending support groups). Due to the minimal number of youth participating in this activity and the fact that 
1(X) percent of the youths’ needs were met, they were included in this footnote and not in the table above. 

*We categorized dropouts as anyone not participating in the traditional school environment. 
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Regarding job-related training and assistance for the 52 dropouts who received services: 

• 33 (63 percent) participated in job preparation classes. 

• 31 (60 percent) received job referrals. 

• 10 (19 percent) participated in work experience activities and/or community service 

activities. 

• 2 (4 percent) participated in their site’s formal youth recruitment program. 

• 4 (8 percent) participated in support groups. 

• 11 (21 percent) participated in recreational activities. 

The overall participation rates for dropouts in job-related training categories appeared to have 
improved as well. Of the 52 youths participating in an activity, only 6 did so in a job 
preparation and/or job referral activity. Most of the youth participated in another work 
experience and/or hard training activity. 

D. Almost half of the enrollees were no longer in contact with the program at the time 
of our fieldwork 

One emphasis of the YOA program was that program staff was supposed to function as 
mentors and counselors rather than routine case managers. For the four final round grantees 
we audited, we determined that approximately 70 percent of the grantees’ expenditures as of 
March 31, 2000, were for (1) administration/ overhead costs and (2) case managers salaries 
and fringe benefits to provide direct mentoring and counseling services to youth. This was not 
as high as the 85 percent we identified in our earlier audit report of the first round YOA 

program. 

Three of the four grantees contracted with community-based organizations (CBOs) to provide 
training courses in either general job preparation subjects or vocational subjects (computer 
repair, etc.). The fourth grantee provided training accounts for youA to attend local 
community colleges or other vocational schools on an as needed basis. As of March 2000, 
direct training costs were 27 percent and support service costs 3 percent of total program 

expenses. 

While there is an emphasis on enrollee counseling/mentoring in the YOA demonstration grants 
WIA Yo! operational grants to raise the youths’ employment possibilities at a liveable 
wage, we believe more should be done to increase direct training assistance. A recent OIG 
audit of the JTPA Title II-C out-of-school youth program found that both participation in, and 
completion of, occupational skills training activities had a significant positive impact on 
youths’ post-program earnings. 

We reviewed case files for the 100 youth in our sample to determine the extent of their contact 
with program counselors. We found that 48 had received bona fide counseling, and 37 were 
not engaged in the program long enough to warrant follow-up services. We defined bona fide 
counseling as evidence that the case manager went a step beyond routine case management 
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(e.g., basic referrals to training). For example, home visits, discussions of personal issues, or 
counseling with third parties on the youth's behalf were included. However, we did not give 
the program credit if the youth did not respond to the counselor’s attempts. Additionally, of 
the 63 youth who were engaged in the program long enough to warrant followup services, 35 
youth (56 percent) received bona fide followup counseling. 

Table 5 below demonstrates whether the case notes for the 100 sample participants indicated 
the counselor had been in recent contact with the youth for any reason. Table 5 does not, 
however, address the quality or substance of the counseling provided. 

Table 5 

Counselor Contact at Time of Fieldwork 
Based on Participants’ Case File Documentation i 



Level of Contact 


Youth Under 18 : 


19-21 Year Olds 


22-24 Year Olds 


total 


In contact with counselor within last 
3 months 


23 


20 


10 


53 


No counselor contact within last 3 
months 


26 


11 


5 


42 


Cannot determine status from case 
fde 


1 


1 


1 


3 


Deceased 


0 


0 


1 


1 


Found to be out of target area 


1 


0 


0 


1 


Total 


51 


32 


17 1 100 



As shown above, 53 youths were still in contact with their case manager/counselors as of the 
time of our fieldwork® (August 2000), and 42 had not had recent contact with their case 
manager/counselors as of that time. 



/ 




^However, 12 of these youth were not participating in an activity. 
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2. In-School Youth Services/Outcomes May Distort Out-of-School Youth 
Demonstration Grants’ Program Performance. 

Of the 100 sampled enrollees, 27 were attending school (in-school) at time of enrollment in the 
OSY demonstration grant program. For our purposes, we defined "in school" as attending 
high school (16 enrollees), attending alternative high school (6 enrollees), attending a GED 
progr am (3 enrollees),'® or attending a post high school vocational/educational program (2 
enrollees) at the time of enrollment. 

ETA’S Request for Proposal (RFP) specified that for these pilot projects Opportunity Areas 
were to be created in target communities to expand employment, education, and traming 
opportunities for out-of-school youth ages 16 through 24, with priority given to high school 
dropouts. The RFP specifically stated: 

Youth employment and development activities . . . shall be used for a structured 
set of initiatives focused sharply on getting out-of-school youth ages 16-24 into 
long-term employment at wage levels that will prevent future dependency. 

[Emphasis added.] 

Because including in-school youths’ activities and outcomes with those out-of-school youth 
may distort the OSY demonstration grants’ results, we analyzed the results of all 27 in-school 
youth independently to determine the activities and services received as well as outcomes 
identified and found that 22" of the 27 youth (81 percent) participated beyond initial 
assessment as follows: 

• 16 (73 percent) attended high school. 

• 15 (68 percent) attended basic skills/GED courses. 

• 3 (14 percent) attended vocational traming courses. 

None of the high school activities appeared to be OSY program-funded. ()ne (3ED program 
appeared to be YOA OSY demonstration funded, but the rest of the activities did not. The 
other vocational training program was an in-school ROTC program. 

Sppr.ial in-school program 

From our discussions with ETA at the exit conference on our first round audit, it was our 
understanding that while in-school youth were not to be denied services under the YOA OSY 



Youth at tpn^iug alternative school and GED programs were also considered dropouts by our defmition and 
included in the dropout statistics reported. 

"some participants participated in more than one activity, so total activities exceeded 22. 
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demonstration grant, they were to be served with nongrant fimds and in-school youth 
performance statistics were not to be included with the intended OSY population. 

Yet, we discovered one of our four audit sites was operating a specific in-school program 
which targeted youth 14 through 15 years of age who were currenUy in school but at risk of 
dropping out. This in-school youth program was approved in their grant. The program 
targeted youth already in the school’s FUTURES program, already in existence prior to the 
YOA OSY demonstration grant iunding. The FUTURES program consisted of an additional 
basic skills course as part of the youths’ high school curriculum and extra counseling/advocate 
attention. The YOA OSY demonstration grant fimded two additional counselors to the 
futures program. 

For this particular site, 13 of the 25 (52 percent) randomly sampled enrollees were participants 
in this special in-school program; consequently, a significant portion of the site’s enrollees as 
of December 31, 1999, were in still in school at the time of enrollment. We attribute this high 
rate of in-school participants to our early sample selection. As of December 31, 1999, the 
grantee had enrolled a disproportionate number of in-school youth due to the ease of 
enrollment. The ETA has informed the OIG that as of December 31, 2000, in-school youth 
represented only 17.5 percent of the grantee’s total enrollments. 

While this in-school program was approved in the grant agreement, the program focus does not 
appear consistent with the purpose of the OSY demonstration grant program. 

In our opinion, performance results reported on the in-school youth could potentially mflate 
performance reporting for the OSY demonstration grant program as a whole; e.g., an 
"attending school" activity could be misinterpreted to indicate the program mfluenced a 
dropout to return to school when the enrollee never left school. 
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3. The YOA OSY Program’s Performance Reporting Structure Has Improved Since 
Our First Round Audit, But Documentation to Support Enrollees’ Activities and 
Outcomes Needs to Be Improved. 

Based on our review of the four grantees’ program management information systems, we 
determined that the grantees had the overall systems in place to report activities and outcomes 
from a mana gement standpoint; however, the level of documentation available to support those 
activities/outcomes was deficient. 



Our main objective in looking at the reporting system was to determine whether there was a 
system in place, including procedures to report activities and outcomes in a traceable, 
preferably automated, format. We did not specifically test the accuracy of the system back to 
our audit sample; as a result, we cannot conclude as to the accuracy of either the information 
reported for our sampled youth or the systems as a whole. 

Three of the four sites had automated systems which reported the youths’ activities and 
outcomes that were traceable to the participant level, and the performance reports submitted to 
ETA appeared to have been traceable to supporting report schedules. The remaming site did 
not have an automated system; however, at the time of our fieldwork they were in the process 
of implementing the client management system supported by ETA. We have subsequently 
been informed that the system is currently operational and is being used for client tracking and 
reporting. 

In addition to determining whether adequate reporting systems were in place, we looked at the 
sufficiency of the documentation maintained to support activities evidenced in the case files. 
Because these were demonstration grants, we accepted as little as a counselor’s note in a file as 
documentation of a participant’s involvement in an activity. However, as the WIA Yo! 
program becomes fully operational, mere references to an activity in a case note will not be 
sufficient; i.e., the level of evidence of enrollees’ activities, including the extent and 
completion of their participation, needs to be increased if the youths’ activities and outcomes 
are to be reported in accordance with the WIA and the Government Performance and Results 
Act (GPRA) reporting standards. 

A. Assessing whether the grantees addressed enrollees’ needs. 

We based our needs analysis in finding 1 on any available case file documentation, since there 
was no requirement for the demonstration grant to conduct an enrollee assessment and develop 
a service strategy. Although there was evidence in the case files that some form of 
assessments were performed for enrollees, the type of evidence varied greatly and was minimal 
in many cases. We acknowledged the enrollees needed the services they were provided, even 
if there was no specific need identified. 
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B. Assessing the level of documentation of participants activities. 

We classified documentation of youths’ participation in activities into the three following 
categories: 

• Indication of participation - any notation that the youth may have attended an 
activity (case notes, MIS form, etc.). 

• Documented attendance but not completion - evidence of attendance such as 
sign-in sheets, attendance reports, or counselors’ notes indicating that they 
observed the enrollees in class. 

• Documented Completion - completion certificate or formal grade report 
evidencing successful completion of activity, or a case manager s file notes 
clearly indicating the youth completed the activity. 

Table 6 categorizes the degree of documentation to support activities in which the sample 
youth participated. 



Table 6 

Level of Documentation Available Supporting Enrollees’ Activities 
Based on Participants’ Case File Documentation 





High 


GED - 


Vocation 


WE/Com 


Job 


Recreat 


Other 




Level of Evidence 


School 


B/S 


Tmg 


Srvc 


Prep 


Act. 


Tmg. 


Total 


Indication of 


41% 


30% 


23% 


69% 


39% 


42% 


90% 


42% 


Participation 


(9) 


(13) 


(7) 


(20) 


(20) 


(11) 


(9) 


(89) 


Documented 


59% 


68% 


51% 


14% 


21% 


23% 


10% 


37% 


Attendance but 


(13) 


(29) 


(16) 


(4) 


(11) 


(6) 


(1) 


(80) 


Not Completion 


















Documented 


0% 


2% 


26% 


17% 


40% 


35% 


0% 


21% 


Completion 


(0) 


(1) 


(8) 


(5) 


(21) 


(9) 


0 


(44) 


Totals 


22 


43 


31 


29 


52 


26 


10 


213 



Only an "indication of participation" was evident in the case files for 42 percent of the 
activities. The high school, GED, and vocational training activities were better documented 
than the work experience, job preparation, or recreation activities. However, also included in 
the high school and GED/Basic Skills statistics were the 12 enrollees who participated in a 
designated in-school program, where the level of documentation was acceptable in most cases. 
In general, for the routine out-of-school programs, the level of documentation was not as 
consistent. Most of the "other training" activities consisted of indication on an MIS form that 
the youth attended counseling support groups, but there was no other mention in the file that 
the youth actually attended the activity. 

er|c 
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For accountability and performance reporting under WIA and GPRA, evidence at an 
"indication of participation" level would not be sufficient to support the activities reported. 

C. Assessing counseling/mentoring documentation. 

Under both YOA demonstration and WIA Yo! operational grants, case management 
(counseling/mentoring) is considered an essential activity rather than an admimstrative function 
of the program. For this reason, the extent to which case management activities are taking 
place should be captured and reported as a separate activity. However, case managers’ salaries 
are classified as training costs along with vocational traming and other related activities. As a 
result, the amount of YOA funds spent on counseling/mentoring versus educational and 
tra inin g activities cannot be separately identified from program reports. 

In many instan ces, the counseling/mentoring contact with youth was not evident in the case 
files. We understand the youth may not be receptive to the counselors’ attempts at contact, and 
that counselors’ have legitimate concerns about maintaining client confidentiality. However, 
counselors’ notes about meetings and discussions with die youth should be sufficient to 
demonstrate the extent and nature of the contacts beyond routine case management. The notes 
should document even unsuccessful attempts at contacting the youth. 

Our limited participant interviews (14 of 100) indicated that more bona fide couMeling and 
mentoring assistance was occurring than was evident in the case files. This may indicate that 
the counselors are hot keeping adequate case notes to fiilly support all the services they are 
providing and for which they should receive credit. 

The YOA OSY demonstration grants we audited were operated under the JTPA. The 
operational Yo! grants are funded by WIA which replaced JTPA. Regardless of the enabling 
legislation, JTPA or WIA, youth programs are ongoing and performance outcomes will 
continue to be reported against established standards. The question to be answered is whether 
the reported outcomes can be validated and relied upon by ETA for GPRA reporting. 

The JTPA, Section 165 provides; 

(a) (1) Recipients shall keep records that are sufficient to permit the 
preparation of reports required by this Act. ... 

(c) Each State, each administrative entity, and each recipient . . . receiving 
funds under this Act shall — 



(2) prescribe and maintain comparable management information systems . . . 
designed to facilitate the uniform compilation, cross tabulation, and analysis of 
programmatic, participant . . . data. . . . 
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(d) (1) The reports required . . . shall include information pertaining to .. . 

(B) the activities in which participants are enrolled, and the 
length of time that participants are engaged in such activities; 

(C) program outcomes . . . for participants. ... 

(e) The Governor shall ensure that requirements are established for retention 
of all records pertinent to all grants awarded . . . including . . . participant 
records and supporting documentation. . . . [Emphasis added.] 

WIA regulation 20 CFR, Part 667.300(e) provides: 

A nnunl Performance Progress Report. An annual performance progress report 
for each of the three programs under title I, subpart B is required by WIA sec. 
136(d). 



(2) States submitting annual performance progress reports that cannot be 
validated or verified as accurately counting and reporting activities in 
accordance with the reporting instructions, may be treated as failing to submit 
annual reports, and be subject to sanctions. . . . [Emphasis added.] 

The GPRA requires that program managers establish performance goals and systems to verify 
and validate the numbers reported. The ETA will be using grantee-reported outcomes to 
report the outcomes of the ftiture WIA Yo! programs for GPRA purposes. The GPRA states: 

(a) In carrying out the provisions of section 1105 (a) (29), the Director of the 
Office of Management and Budget shall require each agency to prepare an 
annual performance plan covering each program activity set forth in the budget 
of such agency. Such plan shall- 

(1) establish performance goals to define the level of performance to be achieved 
by a program activity ... 



(4) establish performance indicators to be used in measuring or assessing the 
relevant outputs, service levels, and outcomes of each program activity; 

(5) provide a basis for comparing actual program results with the established 
performance goals; and 

(6) describe the means to be used to verify and validate measured values. 
[Emphasis added.] 

As these criteria demonstrate, JTPA is specific in requiring that documentation to support 
activities and outcomes be maintained, and both the WIA program regulations and the GPRA 
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emphasize the necessity of having performance outcomes data that can be validated/verified. 
We found the level of evidence to support YOA and future Yo! programs must be improved to 
meet this validation/verification standard. 
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Recommendations 

We recommend the Assistant Secretary for Employment and Training develop a system of 
management controls that provides reasonable assurance that the competitive WIA-fimded 
Youth Opportunity (Yo!) Program will result in substantial and demonstrable educational gains 
and attainment of vocational skills by youth in the targeted areas served by the ETA grants. 
Specifically, we recommend ETA: 

• Develop documentation standards and guidelines for grantees to help educate assigned 
staff and reinforce ETA expectations for minimum case file evidence of youth activities 
and accomplishments/outcomes. The standards should specifically address the level of 
documentation needed to adequately support reporting counseling and mentoring 
activities. 

• Assure that separate performance reporting is maintained for Yo! activities, 
accomplishments, and outcomes for in school as opposed to out-of-school-youth. ETA 
should develop reporting stondards and monitoring procedures to maintain consistent 
supportable performance reporting which we assume is the case for the competitive Yo! 
grants which provide for serving both in-school and out-of-school youth. 

• Issue guidance to all Yo! program grantees encouraging the grantees to: 

• Assess each participant’s needs when enrolled, and as necessary throughout 
program participation. This should be a complete formal assessment, including 
participants input into what needs to be done for them to keep them actively 
enrolled until all substantial needs are met. 

• Provide a wide array of quality services, particularly more vocational training, 
to address the participants’ identified needs, either through WIA funds or other 
available sources (e.g.. Pell grants). 

• Develop and implement performance incentives, including financial incentives, 
if necessary, to high risk dropouts for their major accomplishments such as 
completing high school, obtaining a GED, or successfully completing a 
substantial occupational skills course. 

ETA’S Response to Our Draft Report 

ETA agreed with the need for grantees to better document services participants received and 
indicated ETA would work with the grantees to do so. 
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Specifically ETA responded: 

• ETA will prepare a set of guidelines for documentation levels case managers 
need to adequately support counseling and mentoring activities. 

• The Yo! program MIS currently being implemented separately reports activities 
and outcomes for out-of-school and in-school youth. ETA agrees to develop 
reporting standards and monitoring procedures to maintain supportable 
performance reporting. 

• ETA agrees to develop guidance to the grantees regarding participant 
assessments and services and work with the grantees to develop a set of 
incentives for high school dropouts. 

ETA also reemphasized the importance of the case managers’ involvement in the program as a 
direct service to the participants. 

Finally, ETA indicated that because of the open door policy of providing services to all youth 
in the areas served by the Yo! program, some participants will drop out, and the program may 
have lower performance outcomes than programs that more carefully screen participants. 

Auditor’s Conclusion 

We concur with ETA’s response regarding our recommendations. 

We also concur with ETA’s comments regarding the importance of case managers and the hard 
to serve Yo! program population. However, in our opinion, if the Yo! program participants 
are to attain self-sufficiency ~ the ultimate goal of the program ~ the grantees should find 
ways to use more grant funds to provide specific vocational training. If more direct vocational 
training were provided, the case manager could function more as a counselor/mentor than 
simply case manager, and fewer participants may drop out. 
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DEPUTY ASSISTANT SECRETARY’S 
RESPONSE TO OUR DRAFT REPORT 
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U.S. Department of Labor 



Assistant Secretary for 
Employment and Training 
Washington, D.C. 20210 




MAR 2 0 2001 



MEMORANDUM FOR . 



FROM: 




Comments on OIG Audit of Out-of-School 
Youth Pilot Demonstration Program 



Subject: 



Thank you for providing the Employment and Training Administration (ETA) the opportunity to 
provide comments on your audit of the Out-of-School Youth Pilot Demonstration Program. 

I am pleased that the Office of Inspector General (OIG) has found that the participation rate of 
enrollees has improved from the first-round of grantees-that 89 percent of enrollees in your 
sample have received some type of service and that 74 percent of enrollees received some 
educational or vocational training. I am also pleased that the OIG has foimd that the program’s 
performance reporting structure has improved since your first rovmd audit. The ETA staff has 
spent much time developing and implementing improved MIS systems for these grants. 

I also agree that we need to continue to improve the documentation case managers make in their 
files of services provided to participants. We will work with our grantees to improve this 
documentation. 

The following is om response to each of the specific recommendations in the report as well as 
additional comments about the findings contained in this report. 

RECOMMENDATION: Develop dociunentation standards and guidelines for grantees to help 
educate assigned staff and reinforce the ETA expectations for minimum case file evidence of 
youth activities and accomplishments/outcomes. 

RESPONSE: The ETA agrees to this and will prepare a set of guidelines for the level of 
documentation needed by case managers to adequately support reported counseling and 
mentoring activities. 

RECOMMENDATION: Assure that separate performance reporting is maintained for youth 
opportunity (YO) activities, accomplishments, and outcomes for in-school as opposed to out-of- 
school youth. ETA should develop reporting standards and monitoring procedures to maintain 
consistent supportable performance reporting. 
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RESPONSE: The MIS system which we are currently implementing for YO Grantees does 
separate the activities and outcomes for in-school and out-of-school youth. The ETA agrees to 
develop reporting standards and monitoring procedures to maintain supportable performance 

reporting. 

RECOMMENDATION: Issue guidance to all YO grantees encouraging the grantees to formally 
assess participant’s needs; provide a wide array of quality services, particularly more vocational 
training; and develop performance incentives to high-risk dropouts for completing high school, 

obtaining a GED, or completing a vocational skills course. 

RESPONSE: The ETA agrees to provide such a guidance to YO grantees and to work with 
grantees to develop a set of incentives for high school dropouts. 

There are two points that I would like to make regarding your report: 

First, I want to emphasize that providing case managers to youth is in itself a direct service. 
Researchers in the field of youth programs such as Gary Walker o^ublic/Private Ventures have 
stressed the importance to youth of having at least one adult working closely with them. 
Independently, researchers in the field of juvenile justice such as David Altschuler of the 
University of Maryland are also stressing the importance of having adults working closely with 

youth. 

Second, I want to emphasize that the nature of the Youth Opportunity Grants is to do outreach 
and serve all interested youth living within the high-poverty target communities. It would be 
contradictory to the purpose of these grants to screen all applicants and only serve the most 
motivated or job-ready. By doing outreach to serve all youth, we necessarily are going to lose 
some youth and have lower outcomes than programs that carefully screen and accept only 
motivated or youth with reading levels about a certain level. 

Thank you again for working with us in issuing this report and for providing this opportunity to 
comment. 
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